On the Complexity of Learning Decision Trees

Various factors a ecting decision tree learning time are explored. The factors which consistently a ect accuracy are those which directly or indirectly (as in the handling of continuous attributes) allow a greater variety of potential trees to be explored. Other factors, e.g., pruning and choice of heuristics, generally have little e ect on accuracy, but signi cantly a ect learning time. We prove that the time complexity of induction and post-processing is exponential in tree height in the worst case and, under fairly general conditions, in the average case. This puts a premium on designs which produce shallower and more balanced trees. Simple pruning is linear in tree height, contrasted to the exponential growth of more complex operations. The key factor in uencing whether simple pruning will su ce is that the split selection and pruning heuristics should be the same and unbiased. The information gain and 2 tests are biased towards unbalanced splits, and neither is admissible for pruning. Empirical results show that the hypergeometric function can be used for both split selection and pruning, and that the resulting trees are simpler, more quickly learned, and no less accurate than trees resulting from other heuristics and more complex post-processing.